fix(inference): unpack structured records per sample, not all-or-nothing - #1669

Fixes #1665.

The bug

infer_data_job() splits the structured return shape into prediction /
extra_records columns only if every sample is structured:

if all(_is_structured_record(data_all[x]) for x in data['index']):

A failed sample can never satisfy that predicate. BaseAPI.generate returns
the structured dict only on the success path, and falls through to a plain
string once retries are exhausted:

if ret_code == 0 and self.fail_msg not in answer and answer != '':
    if isinstance(log, dict):
        return {"prediction": answer, "extra_records": log}
    return answer
...
return self.fail_msg if answer in ['', None] else answer   # <- plain str

So one transient API failure anywhere in the run flips all(...) to False
and every successful structured record is passed through str().
vlmeval/api/arm_thinker.py is a shipped model class that hits this.

The fix

Unpack per sample through a shared helper, and gate on any(...). A failed
row keeps its fail_msg string in prediction (so failure accounting and
retry_failed are unaffected) and gets {} in extra_records.

The SPLIT_THINK branch had the identical gate and is fixed the same way.

Verification

verify.py lifts the real rank-0 assembly block out of vlmeval/inference.py
with ast (node.lineno..end_lineno, then dedent) from both the main
version and the patched version, so nothing is hand-copied, and replays it
over a 3-sample run where sample 2 exhausted its retries.

Mixed run — the reported case:

================ SPLIT_THINK unset ================
-- BEFORE (main)
   columns      : ['index', 'prediction']
   prediction[0]: "{'prediction': '<think>t0</think>A', 'extra_records': {'tool_call_count': 1}}"
   prediction[1]: "{'prediction': '<think>t1</think>B', 'extra_records': {'tool_call_count': 2}}"
   prediction[2]: 'Failed to obtain answer via API.'
   extra_records: <COLUMN MISSING>
-- AFTER  (patch)
   columns      : ['index', 'prediction', 'extra_records']
   prediction[0]: '<think>t0</think>A'
   prediction[1]: '<think>t1</think>B'
   prediction[2]: 'Failed to obtain answer via API.'
   extra_records: [{'tool_call_count': 1}, {'tool_call_count': 2}, {}]

================ SPLIT_THINK=1 ================
-- BEFORE (main)
   prediction[0]: "A', 'extra_records': {'tool_call_count': 1}}"
   prediction[1]: "B', 'extra_records': {'tool_call_count': 2}}"
   extra_records: <COLUMN MISSING>
-- AFTER  (patch)
   prediction[0]: 'A'
   prediction[1]: 'B'
   extra_records: [{'tool_call_count': 1}, {'tool_call_count': 2}, {}]

Regression guard — homogeneous runs must not move:

IDENTICAL  all structured (no failures) / SPLIT_THINK unset
IDENTICAL  all structured (no failures) / SPLIT_THINK=1
IDENTICAL  none structured (plain string model) / SPLIT_THINK unset
IDENTICAL  none structured (plain string model) / SPLIT_THINK=1

all homogeneous cases unchanged: True

pre-commit's flake8 (7.1.2, --max-line-length=120 --ignore=W503) reports
the same findings before and after, none of them on an added line; isort 6.0.1
is clean.

🤖 Generated with Claude Code